Piecewise Deterministic Markov Processes Research Articles

“PEOPLE” MEET “MARKOVIANS” — INDIVIDUAL-BASED MODELING WITH HYBRID STOCHASTIC SYSTEMS

Individual-based models (IBMs) enable modelers to avoid far-reaching abstractions and strong simplifications by allowing for a state-based representation of individuals. The fact that IBMs are not represented using a standardized mathematical framework such as differential equations makes it harder to reproduce IBMs and introduces difficulties in the analysis of IBMs. We propose a model architecture based on representing individuals via Markov models. Individuals are coupled to populations — for which individuals are not explicitly represented — that are modeled by differential equations. The resulting models consisting of continuous-time finite-state Markov models coupled to systems of differential equations are examples of piecewise-deterministic Markov processes (PDMPs). We will demonstrate that PDMPs, also known as hybrid stochastic systems, allow us to design detailed state-based representations of individuals which, at the same time, can be systematically analyzed by taking advantage of the theory of PDMPs. We will illustrate design and analysis of IBMs using PDMPs via the example of a predator that intermittently feeds on a logistically growing prey by stochastically switching between a resting and a feeding state. This simple model shows a surprisingly rich dynamics which, nevertheless, can be comprehensively analyzed using the theory of PDMPs.

Stochastic modeling of a gene regulatory network driving B cell development in germinal centers.

Germinal centers (GCs) are the key histological structures of the adaptive immune system, responsible for the development and selection of B cells producing high-affinity antibodies against antigens. Due to their level of complexity, unexpected malfunctioning may lead to a range of pathologies, including various malignant formations. One promising way to improve the understanding of malignant transformation is to study the underlying gene regulatory networks (GRNs) associated with cell development and differentiation. Evaluation and inference of the GRN structure from gene expression data is a challenging task in systems biology: recent achievements in single-cell (SC) transcriptomics allow the generation of SC gene expression data, which can be used to sharpen the knowledge on GRN structure. In order to understand whether a particular network of three key gene regulators (BCL6, IRF4, BLIMP1), influenced by two external stimuli signals (surface receptors BCR and CD40), is able to describe GC B cell differentiation, we used a stochastic model to fit SC transcriptomic data from a human lymphoid organ dataset. The model is defined mathematically as a piecewise-deterministic Markov process. We showed that after parameter tuning, the model qualitatively recapitulates mRNA distributions corresponding to GC and plasmablast stages of B cell differentiation. Thus, the model can assist in validating the GRN structure and, in the future, could lead to better understanding of the different types of dysfunction of the regulatory mechanisms.

PDMP Monte Carlo methods for piecewise smooth densities

Abstract There has been substantial interest in developing Markov chain Monte Carlo algorithms based on piecewise deterministic Markov processes. However, existing algorithms can only be used if the target distribution of interest is differentiable everywhere. The key to adapting these algorithms so that they can sample from densities with discontinuities is to define appropriate dynamics for the process when it hits a discontinuity. We present a simple condition for the transition of the process at a discontinuity which can be used to extend any existing sampler for smooth densities, and give specific choices for this transition which work with popular algorithms such as the bouncy particle sampler, the coordinate sampler, and the zigzag process. Our theoretical results extend and make rigorous arguments that have been presented previously, for instance constructing samplers for continuous densities restricted to a bounded domain, and we present a version of the zigzag process that can work in such a scenario. Our novel approach to deriving the invariant distribution of a piecewise deterministic Markov process with boundaries may be of independent interest.

Adaptive Importance Sampling Based on Fault Tree Analysis for Piecewise Deterministic Markov Process

Adaptive Importance Sampling Based on Fault Tree Analysis for Piecewise Deterministic Markov Process

Necessary and sufficient symmetries in Event-Chain Monte Carlo with generalized flows and application to hard dimers.

Event-Chain Monte Carlo (ECMC) methods generate continuous-time and non-reversible Markov processes, which often display significant accelerations compared to their reversible counterparts. However, their generalization to any system may appear less straightforward. In this work, our aim is to distinctly define the essential symmetries that such ECMC algorithms must adhere to, differentiating between necessary and sufficient conditions. This exploration intends to delineate the balance between requirements that could be overly limiting in broad applications and those that are fundamentally essential. To do so, we build on the recent analytical description of such methods as generating piecewise deterministic Markov processes. Therefore, starting with translational flows, we establish the necessary rotational invariance of the probability flows, along with determining the minimum event rate. This rate is identified with the corresponding infinitesimal Metropolis rejection rate. Obeying such conditions ensures the correct invariance for any ECMC scheme. Subsequently, we extend these findings to encompass schemes involving deterministic flows that are more general than mere translational ones. Specifically, we define two classes of interest of general flows: the ideal and uniform-ideal ones. They, respectively, suppress or reduce the event rates. From there, we implement a comprehensive non-reversible sampling of a system of hard dimers by introducing rotational flows, which are uniform-ideal. This implementation results in a speed-up of up to ∼3 compared to the state-of-the-art ECMC/Metropolis hybrid scheme.

NuZZ: Numerical Zig-Zag for general models

Markov chain Monte Carlo (MCMC) is a key algorithm in computational statistics, and as datasets grow larger and models grow more complex, many popular MCMC algorithms become too computationally expensive to be practical. Recent progress has been made on this problem through development of MCMC algorithms based on Piecewise Deterministic Markov Processes (PDMPs), irreversible processes which can be engineered to converge at a rate which is independent of the size of the dataset. While there has understandably been a surge of theoretical studies following these results, PDMPs have so far only been implemented for models where certain gradients can be bounded in closed form, which is not possible in many relevant statistical problems. Furthermore, there has been substantionally less focus on practical implementation, or the efficiency of PDMP dynamics in exploring challenging densities. Focusing on the Zig-Zag process, we present the Numerical Zig-Zag (NuZZ) algorithm, which is applicable to general statistical models without the need for bounds on the gradient of the log posterior. This allows us to perform numerical experiments on: (i) how the Zig-Zag dynamics behaves on some test problems with common challenging features; and (ii) how the error between the target and sampled distributions evolves as a function of computational effort for different MCMC algorithms including NuZZ. Moreover, due to the specifics of the NuZZ algorithms, we are able to give an explicit bound on the Wasserstein distance between the exact posterior and its numerically perturbed counterpart in terms of the user-specified numerical tolerances of NuZZ.

A decentralized algorithm for a mean field control problem of piecewise deterministic Markov processes

This paper provides a decentralized approach for the control of a population of N agents to minimize an aggregate cost. Each agent evolves independently according to a Piecewise Deterministic Markov dynamics controlled via unbounded jumps intensities. The N-agent high dimensional stochastic control problem is approximated by the limiting mean field control problem. A Lagrangian approach is proposed. Although the mean field control problem is not convex, it is proved to achieve zero duality gap. A stochastic version of the Uzawa algorithm is shown to converge to the primal solution. At each dual iteration of the algorithm, each agent solves its own small dimensional sub problem by means of the Dynamic Programming Principal, while the dual multiplier is updated according to the aggregate response of the agents. Finally, this algorithm is used in a numerical simulation to coordinate the charging of a large fleet of electric vehicles in order to track a target consumption profile.

Recent advances in the long-time analysis of killed degenerate processes and their particle approximation

We review some recent results of quantitative long-time convergence for the law of a killed Markov process conditioned to survival toward a quasi-stationary distribution, and on the analogous question for the particle systems used in practice to sample these distributions. With respect to the existing literature, one of the novelties of these works is the degeneracy of the underlying process with respect to classical elliptic diffusion, namely it can be a non-elliptic hypoelliptic diffusion, a piecewise deterministic Markov process or an Euler numerical scheme.

The Central Limit Theorem for Markov Processes that are Exponentially Ergodic in the Bounded-Lipschitz Norm

In this paper, we establish a version of the central limit theorem for Markov–Feller continuous time processes (with a Polish state space) that are exponentially ergodic in the bounded-Lipschitz distance and enjoy a continuous form of the Foster–Lyapunov condition. As an example, we verify the assumptions of our main result for a specific piecewise-deterministic Markov process, whose deterministic component evolves according to continuous semiflows, switched randomly at the jump times of a Poisson process.

Infinite dimensional Piecewise Deterministic Markov Processes

In this paper we aim to construct infinite dimensional versions of well established Piecewise Deterministic Monte Carlo methods, such as the Bouncy Particle Sampler, the Zig-Zag Sampler and the Boomerang Sampler. In order to do so we provide an abstract infinite dimensional framework for Piecewise Deterministic Markov Processes (PDMPs) with unbounded event intensities. We further develop exponential convergence to equilibrium of the infinite dimensional Boomerang Sampler, using hypocoercivity techniques. Furthermore we establish how the infinite dimensional Boomerang Sampler admits a finite dimensional approximation, rendering it suitable for computer simulation.

Posterior Computation with the Gibbs Zig-Zag Sampler

An intriguing new class of piecewise deterministic Markov processes (PDMPs) has recently been proposed as an alternative to Markov chain Monte Carlo (MCMC). We propose a new class of PDMPs termed Gibbs zig-zag samplers, which allow parameters to be updated in blocks with a zig-zag sampler applied to certain parameters and traditional MCMC-style updates to others. We demonstrate the flexibility of this framework on posterior sampling for logistic models with shrinkage priors for high-dimensional regression and random effects, and provide conditions for geometric ergodicity and the validity of a central limit theorem.

Pricing of Credit Risk Derivatives with Stochastic Interest Rate

This paper deals with a credit derivative pricing problem using the martingale approach. We generalize the conventional reduced-form credit risk model for a credit default swap market, assuming that the firms’ default intensities depend on the default states of counterparty firms and that the stochastic interest rate follows a jump-diffusion Cox–Ingersoll–Ross process. First, we derive the joint Laplace transform of the distribution of the vector process (rt,Rt) by applying piecewise deterministic Markov process theory and martingale theory. Then, using the joint Laplace transform, we obtain the explicit pricing of defaultable bonds and a credit default swap. Lastly, numerical examples are presented to illustrate the dynamic relationships between defaultable securities (defaultable bonds, credit default swap) and the maturity date.

Exponential ergodicity for damping Hamiltonian dynamics with state-dependent and non-local collisions

In this paper, we investigate the exponential ergodicity in a Wasserstein-type distance for a damping Hamiltonian dynamics with state-dependent and non-local collisions, which indeed is a special case of piecewise deterministic Markov processes that is very popular in numerous modelling situations including stochastic algorithms. The approach adopted in this work is based on a combination of the refined basic coupling and the refined reflection coupling for non-local operators. In a certain sense, the main result developed in the present paper is a continuation of the counterpart in (Stochastic Process. Appl. (2022) 146 114–142) on exponential ergodicity of stochastic Hamiltonian systems with Lévy noises and a complement of (Ann. Inst. Henri Poincaré Probab. Stat. 58 (2022a) 916–944) upon exponential ergodicity for Andersen dynamics with constant jump rate functions.

Piecewise deterministic Markov process for condition-based imperfect maintenance models

Piecewise deterministic Markov process for condition-based imperfect maintenance models

Output regulation of stochastic sampled-data systems with post-processing internal model

This paper deals with the output regulation problem (ORP) of a linear time-invariant (LTI) system in the presence of sporadically sampled measurement streams with the inter-sampling intervals following a stochastic process. Under such sporadically available measurement streams, a regulator consisting of a hybrid observer, continuous-time post-processing internal model, and stabilizer are proposed, which resets with the arrival of new measurements. The resulting system exhibits a deterministic behavior except for the jumps that occur at random sampling times and therefore the overall closed-loop system can be categorized as a piecewise deterministic Markov process (PDMP). In existing works on ORPs with aperiodic sampling, the requirement of boundedness on inter-sampling intervals precludes extending the solution to the random sampling intervals with possibly unbounded support. Using the Lyapunov-like theorem for the stability analysis of stochastic systems, we offer sufficient conditions to ensure that the overall closed-loop system is mean exponentially stable (MES) and the objectives of the ORP are achieved under stochastic sampling of measurement streams. The resulting LMI conditions lead to a numerically tractable design of the hybrid regulator. Finally, with the help of an illustrative example, the effectiveness of the theoretical results are verified.

Adaptive discounted control for piecewise deterministic Markov processes

The main goal of this paper is to study the adaptive infinite-horizon discounted continuous-time optimal control problem of piecewise deterministic Markov processes (PDMPs) with the control acting continuously on the jump intensity λ and on the transition measure Q of the process. It is assumed that jump parameters (λ and Q), as well the continuous and boundary costs (Cg and Ci respectively), depend on an unknown parameter β⁎. It is shown that the principle of estimation and control holds, that is, the strategy consisting of choosing, at each stage n, an action according to an optimal stationary policy, where the true but unknown parameter β⁎ is replaced by its estimated value βˆn, is asymptotically discount optimal, provided that the sequence of estimators {βˆn} of β⁎ is strongly consistent, that is, βˆn converge to β⁎ almost surely. In the framework of PDMPs, the so-called discrepancy function depends on the derivative along the flow of the value function as well as on some boundary conditions, which brings new challenges in the analysis of this problem.

Concave-Convex PDMP-based Sampling

Recently nonreversible samplers based on simulating piecewise deterministic Markov processes (PDMPs) have shown potential for efficient sampling in Bayesian inference problems. However, there remains a lack of guidance on how to best implement these algorithms. If implemented poorly, the computational costs of simulating event times can outweigh the statistical efficiency of the nonreversible dynamics. Drawing on the adaptive rejection literature, we propose the concave-convex adaptive thinning approach for simulating a piecewise deterministic Markov process, which we call CC-PDMP. This approach provides a general guide for constructing bounds that may be used to facilitate PDMP-based sampling. A key advantage of this method is its additive structure—adding concave-convex decompositions yields a concave-convex decomposition. This makes the construction of bounds modular, as given a concave-convex decomposition for a class of likelihoods and a family of priors, they can be combined to construct bounds for the posterior. We show that constructing our bounds is simple and leads to computationally efficient thinning. Our approach is well suited to local PDMP simulation where conditional independence of the target can be exploited for potentially huge computational gains. We provide an R package and compare with existing approaches to simulating events in the PDMP literature. Supplementary materials for this article are available online.

Averaging for slow–fast piecewise deterministic Markov processes with an attractive boundary

AbstractIn this paper we consider the problem of averaging for a class of piecewise deterministic Markov processes (PDMPs) whose dynamic is constrained by the presence of a boundary. On reaching the boundary, the process is forced to jump away from it. We assume that this boundary is attractive for the process in question in the sense that its averaged flow is not tangent to it. Our averaging result relies strongly on the existence of densities for the process, allowing us to study the average number of crossings of a smooth hypersurface by an unconstrained PDMP and to deduce from this study averaging results for constrained PDMPs.

PDMP Characterisation of Event-Chain Monte Carlo Algorithms for Particle Systems

Monte Carlo simulations of systems of particles such as hard spheres or soft spheres with singular kernels can display around a phase transition prohibitively long convergence times when using traditional Hasting–Metropolis reversible schemes. Efficient algorithms known as event-chain Monte Carlo (ECMC) were then developed to reach necessary accelerations. They are based on non-reversible continuous-time Markov processes. Proving invariance and ergodicity for such schemes cannot be done as for discrete-time schemes and a theoretical framework to do so was lacking, impeding the generalisation of ECMC algorithms to more sophisticated systems or processes. In this work, we characterize the Markov processes generated in ECMC as piecewise deterministic Markov processes. It first allows us to propose more general schemes, for instance regarding the direction refreshment. We then prove the invariance of the correct stationary distribution. Finally, we show the ergodicity of the processes in soft- and hard-sphere systems, with a density condition for the latter.

The Markovian Shot-noise Risk Model: A Numerical Method for Gerber-Shiu Functions

In this paper, we consider discounted penalty functions, also called Gerber-Shiu functions, in a Markovian shot-noise environment. At first, we exploit the underlying structure of piecewise-deterministic Markov processes (PDMPs) to show that these penalty functions solve certain partial integro-differential equations (PIDEs). Since these equations cannot be solved exactly, we develop a numerical scheme that allows us to determine an approximation of such functions. These numerical solutions can be identified with penalty functions of continuous-time Markov chains with finite state space. Finally, we show the convergence of the corresponding generators over suitable sets of functions to prove that these Markov chains converge weakly against the original PDMP. That gives us that the numerical approximations converge to the discounted penalty functions of the original Markovian shot-noise environment.